Image processing device, image processing method, and image processing program

ABSTRACT

An object plane is set, and appropriate image processing is performed according to the setting of the object plane. An image processing apparatus has a controller ( 7 ). The controller ( 7 ) receives image data for a captured image and distance information on the distance to a designated point in the captured image, sets an object plane ( 100 ) containing the designated point, converts the distance information to the information on the distance from the object plane, and performs predetermined image processing according to the converted distance information.

CROSS-REFERENCES TO RELATED APPLICATIONS

The present application is a reissue application of U.S. Pat. No.8,564,693 which issued on Oct. 22, 2013 which is a national phase entryunder 35 U.S.C. § 371 of International Application No. PCT/JP2008/060733filed Jun. 5, 2008, published on Dec. 11, 2008 as WO 2008/150027 A1,which claims priority from Japanese Patent Application No. JP2007-150071 filed in the Japanese Patent Office on Jun. 6, 2007, andJapanese Patent Application No. JP 2008-133498 filed in the JapanesePatent Office on May 21, 2008.

TECHNICAL FIELD

The present invention relates to an image processing apparatus and, moreparticularly, to an image processing apparatus, image processing methodand image processing program for performing image processing, such asblurring, according to a set object plane.

BACKGROUND ART

With a conventional imaging apparatus such as a film camera or digitalcamera, imaging may be performed by using camera movement of an imaginglens. “Camera movement” means lens operation (or camera operation) forchanging imaging effect by breaking the basic relation between theimaging apparatus and the imaging lens in which the optical axis of thelens is perpendicular to the film plane (or imaging device plane) at thecenter of the imaging screen. Such operations are collectively called“camera movement”. Camera movement allows the perspective deformation ofbuildings or the like to be corrected, the depth of field to becontrolled, and the camera's reflection in a mirror or the like to beavoided.

Camera movement can be broadly classified as shown in FIG. 35.

Camera movement includes two groups of operations. One is “displacement”including “shift” and “rise/fall”. The other includes “swing” and“tilt”. The former translates the lens plane (or the film plane/imagingdevice plane) while maintaining the perpendicular relation between theoptical axis of the imaging lens and the film plane (or the imagingdevice plane). In other words, the position at which the optical axis isperpendicular to the screen is translated off the center of the screen.The latter breaks the perpendicular relation between the optical axis ofthe imaging lens and the film plane (or the imaging device plane). Inother words, the crossed angle between the optical axis and the filmplane (or the imaging device plane) are changed from 90°.

In a strict sense, only the former “displacement” corresponds to cameramovement. However, in a general sense, both the former and latter areoften confused and collectively called “camera movement”. Then, in thefollowing, for convenience, “shift,” “rise/fall,” “tilt,” and “swing”are called shift movement, rise/fall movement, tilt movement, and swingmovement, respectively.

These are described in more detail below. “Shift movement” means movingthe optical axis of the lens (or the film plane/the imaging deviceplane) left or right, which is used for improving composition orcorrecting distortion. This operation allows changing the composition inthe horizontal direction without moving the camera itself. For example,using shift movement, it is possible to perform imaging at the positionshifted left or right from the center of a mirror and obtain aphotograph as if taken at the center of the mirror. “Rise/fall movement”means moving the lens plane (or the film plane/the imaging device plane)up or down, which is also used for improving composition or correctingdistortion as with shift movement. Rise/fall movement allows improvingcomposition in the vertical direction without extending the elevator orleg of a tripod. When shooting looking up at a tall building with anormal camera, the far part of the building may appear smaller andtapered in a photograph. Rise/fall movement may be used for perspectivecontrol to shoot the building so that it will appear upright.

“Tilt movement” means tilting the lens plane (or the film plane/theimaging device plane) forwards or backwards with respect to the opticalaxis of the lens, which is used generally for changing the position ofthe plane of sharp focus or correcting distortion. Tilt movement allowsfocusing over the wide range from near field to far field even at fullaperture. “Swing movement” means swinging the lens plane (or the filmplane/the imaging device plane) on the axis perpendicular to the opticalaxis of the imaging lens, which effect is similar to that of tiltmovement described above.

For example, one principle for making depth of field look deeper bymanipulating the object plane using tilt movement is known as“Scheimpflug principle”. In the following, an imaging method using theScheimpflug principle is briefly described with reference to FIG. 36.The Scheimpflug principle says that, generally, three planes extendedfrom the object plane, the lens plane, and the imaging plane intersectatone line. Specifically, according to this principle, when an imagingdevice plane 400 and the principal plane of an imaging lens 401 aretilted to each other, an object plane 402 tilted so as to intersect withthe imaging device plane 400 and the principal plane of the imaging lens401 at one ridge line will be brought into sharp focus. In anotherrecently known imaging method, the object plane is changed by adjustingthe tilt angle with respect to the optical axis of the imaging lens 401.

In yet another known imaging method, for example, by using the cameramovement effect according to the above method, towns and buildings areshot as if they were miniatures by adjusting the object plane, inaddition to making depth of field look deeper by manipulating the objectplane.

The above described imaging methods provide camera movement effects bymanipulating an optical system. On the other hand, examples of imagingmethods providing camera movement effects by digital processing includean imaging apparatus for performing correction by camera movement (shiftmovement, rise/fall movement) using image processing, without using aspecial lens (see JP-A-8-163428, for example), and a digital camera forconverting an image that is already given a camera movement effect (forexample, an image of a white board shot from an obliquely lowerposition) to a pseudo front image based on a manually set tilt angle andranging data to an object (see JP-A-9-289610).

However, the technology disclosed in JP-A-8-163428 provides geometriccorrection similar to camera movement effect electronically with digitaltechnology, but does not provide an imaging method corresponding toswing movement and tilt movement using image processing.

Also, the technology disclosed in JP-A-9-289610 provides a method forimaging by converting an image shot from an oblique direction to apseudo front image, but does not provide an imaging method correspondingto swing movement and tilt movement using image processing, as inJP-A-8-163428.

In addition, obtaining by image processing an image that is given aneffect equal to that of imaging with such a shallow depth of field andtilted object plane that imaging may not be performed with an actualimaging device/focal length, and obtaining by image processing an imagethat is given an effect equal to that of imaging with shallow depth offield have not been achieved yet.

DISCLOSURE OF THE INVENTION

In view of the above, the present invention is directed to perform imageprocessing, such as blurring, according to a set object plane.

An image processing apparatus in accordance with a first aspect of theinvention includes: an object plane setting section for setting anobject plane based on distance information corresponding to a pointdesignated by a user in a displayed image based on image data; adistance information conversion section for converting the distanceinformation of the image data according to the object plane; and animage processing section for performing image processing on the imagedata based on the distance information converted by the distanceinformation conversion section.

An image processing method in accordance with a second aspect of theinvention includes the step of: setting an object plane based ondistance information corresponding to a point designated by a user in adisplayed image based on image data; converting the distance informationof the image data according to the object plane; and performing imageprocessing on the image data based on the converted distanceinformation.

An image processing program in accordance with a third aspect of theinvention causes a computer to execute an image processing methodcomprising the step of: setting an object plane based on distanceinformation corresponding to a point designated by a user in a displayedimage based on image data; converting the distance information of theimage data according to the object plane; and performing imageprocessing on the image data based on the converted distanceinformation.

Thus, according to the first to third aspects, an object plane is set,and appropriate image processing is performed according to the settingof the object plane.

According to the invention, an image processing apparatus, imageprocessing method and image processing program for performing imageprocessing, such as blurring, according to the set object plane can beprovided.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a conceptual drawing showing the basic principle of an imageprocessing apparatus in accordance with a first embodiment of theinvention.

FIGS. 2(a) and (b) are conceptual drawings illustrating the setting ofan object plane.

FIG. 3 shows a configuration of the image processing apparatus inaccordance with the first embodiment of the invention.

FIG. 4 is a flowchart describing in detail a characteristic processperformed by the image processing apparatus in accordance with the firstembodiment of the invention to provide a correction equal to cameramovement effect.

FIG. 5 is a flowchart describing the temporary setting of the objectplane.

FIG. 6 is a flowchart describing the adjustment of the object plane.

FIG. 7(a) shows an example of displaying a captured image.

FIG. 7(b) shows an example of displaying a depth map (distanceinformation).

FIG. 7(c) shows an example of converting the depth map to theinformation on the distance from the object plane.

FIG. 7(d) shows an example of displaying a blurred image.

FIG. 8 is a conceptual drawing describing the conversion of the distanceinformation performed along with changing the object plane.

FIG. 9 is a conceptual drawing describing the basic principle ofobtaining the distance information.

FIGS. 10(a) to (c) are conceptual drawings describing an example ofsetting the object plane to any curved surface.

FIG. 11 shows an example of displaying with a window for showing theobject plane.

FIG. 12 shows a configuration of an information processing apparatus inaccordance with a second embodiment of the invention.

FIG. 13 shows a configuration of an image processing apparatus inaccordance with a third embodiment of the invention.

FIGS. 14(a) and (b) are perspective views of the image processingapparatus in accordance with the third embodiment.

FIG. 15 is a flowchart describing in detail a characteristic processusing the image processing apparatus in accordance with the thirdembodiment of the invention.

FIG. 16 is a flowchart describing in detail a process relating to themanipulation of the object plane.

FIGS. 17(a) and (b) show an example of displaying for describing themanipulation of the object plane.

FIG. 18 is a flowchart describing in detail a process relating to theimage processing.

FIG. 19 shows a relation between the blurring parameter and the distanceinformation.

FIGS. 20(a) to (c) show examples of displaying images relating toundoing.

FIGS. 21(a) to (c) show examples of displaying images relating tosetting the depth of field in a fourth embodiment of the invention.

FIGS. 22(a) to (d) show examples of displaying images relating tosetting and finishing setting the depth of field in a fifth embodimentof the invention.

FIGS. 23(a) and (b) show examples of displaying images relating tosetting the object plane in the fifth embodiment of the invention.

FIG. 24 shows an example of displaying the amount of tilt of the objectplane in a sixth embodiment of the invention.

FIGS. 25(a) to (c) show examples of displaying horizontal and verticallines and the like for fixing the object plane in a seventh embodimentof the invention.

FIGS. 26(a) to (h) show examples of displaying the object plane in aneighth embodiment of the invention.

FIG. 27(a) is a conceptual drawing showing a meridional image plane.

FIG. 27(b) is a conceptual drawing showing a sagittal image plane.

FIG. 28 shows the shape of the object plane represented like the depthmap.

FIG. 29 conceptually shows the shape of the object plane.

FIGS. 30(a) and (b) show examples of displaying images relating tochanging the shape of the object plane in a ninth embodiment of theinvention.

FIGS. 31(a) and (b) show examples of displaying images relating tosetting the object plane in a tenth embodiment of the invention.

FIGS. 32(a) and (b) show examples of displaying other images relating tosetting the object plane in the tenth embodiment of the invention.

FIGS. 33(a) to (c) show examples of displaying other images relating tosetting the object plane in the tenth embodiment of the invention.

FIGS. 34(a) to (c) show examples of displaying images relating tounsetting the object plane in the tenth embodiment of the invention.

FIG. 35 shows various types of camera movement.

FIG. 36 is a conceptual drawings illustrating Scheimpflug principle.

BEST MODE FOR CARRYING OUT THE INVENTION

The best mode for carrying out the present invention (simply referred toas embodiments hereinafter) is described in detail below with referenceto the drawings.

(First Embodiment)

First, the basic principle of an image processing apparatus inaccordance with a first embodiment of the invention is described withreference to a conceptual drawing shown in FIG. 1. In this figure, theimage processing apparatus in accordance with the first embodiment isexemplarily applied to a digital camera as an imaging apparatus, but isnot limited thereto.

As shown in FIG. 1, when imaging is performed by an imaging apparatus 1,predetermined image processing is performed, and then image data anddistance information (distance data) are stored in an internal memory 2.The image data is read out as appropriate and the captured image isdisplayed on a display 3 such as a liquid crystal display (LCD).

While looking at the captured image displayed on the display 3, a userdesignates one or more designated points (desired points) by operating atouch panel 5, and arbitrarily sets the tilt direction and the tiltangle of an object plane by operating a cross-shaped key 4 (consistingof Up, Down, Left, and Right buttons, for example).

In this example, the internal memory 2 provides a storage capability,for example. The touch panel 5 provides a designation capability, forexample. The cross-shaped key 4 provides a setting capability, forexample.

In this configuration, a controller 7 sets the object plane based on thedistance information for each region of the displayed image according tothe image data, and based on the distance information of the designatedpoint (s) designated by the user in the displayed image. Then thecontroller 7 converts the distance information for each point related tothe image data, according to the object plane. An image processor 6performs image processing on the image data based on the converteddistance information. Thus, in this configuration, the controller 7provides an object plane setting capability and a distance informationconversion capability, and the image processor 6 provides an imageprocessing capability.

Specifically, as shown in FIG. 2(a), when the user designates one pointon the touch panel 5, a plane that contains the designated pointdesignated by the user and is perpendicular to the optical axis of animaging lens (optical axis of light incident upon an imager) istemporarily set as object plane 100, and then, the tilt direction andtilt angle of the object plane 100 become able to be set by rotating theobject plane 100 on the X-axis and on the Y-axis, the two axes crossingat the designated point. The designated point is a point contained inthe main object to be photographed.

For example, when the Left or Right button of the cross-shaped key 4 isoperated, the object plane 100 rotates on the Y-axis, and the tiltdirection and tilt angle corresponding to the rotation are set. Also,for example, when the Up or Down button of the cross-shaped key 4 isoperated, the object plane 100 rotates on the X-axis, and the tiltdirection and tilt angle corresponding to the rotation are set. As anexample, FIG. 2(b) shows an object plane 101, in which an operation ofthe cross-shaped key 4 cause a rotation on the Y-axis to set thedirection of the tilt around the Y-axis and to set the angle of the tiltto θ shown in the figure. However, rotating is not necessary. The objectplane 100 can be set only by designating one point even withoutrotating.

When the desired setting of the object plane 100 (101) using the touchpanel 5 and the cross-shaped key 4 is completed, an image on whichpredetermined image processing such as blurring, is performed by theimage processor 6 in the imaging apparatus are displayed on the display3 (or an external display not shown).

More specifically, in this example, the distance information for eachpoint of the image data are converted based on the coordinates of thedesignated point, then image processing such as blurring is performed onthe image data based on the converted distance information. When theobject plane is changed by rotating the object plane on the designatedpoint according to the user designation, the distance information foreach point of the image data is converted based on the coordinates ofthe designated point and the angle of the rotation. Then imageprocessing such as blurring is performed on the image data based on theconverted distance information. In this example, the image processor 6provides this image processing capability, for example.

The distance information used in this embodiment is what is called depthmap, which is the individual distance data for one or more points in thecaptured image. It should be noted that, when an image is most finelydivided into regions, each region corresponds to a pixel. In this case,depth map data could mean distance data for each of a plurality ofpixels.

Methods for obtaining this distance information include TOF (time offlight) method and depth from defocus method. TOF method is a techniquefor determining the distance to an object based on the delay time duringwhich a light is emitted from a light source, is reflected by theobject, and reaches a sensor, and the speed of light. These methods aredescribed in detail later.

As described above, for the image processing apparatus in accordancewith the first embodiment, when one point in the captured image isdesignated, a plane that contains the designated point and isperpendicular to the optical axis is set as object plane. Morespecifically, a plane that contains the designated point and a pointhaving the same distance information as that of the designated point isset as object plane.

Furthermore, for example, when two or more points contained in thecaptured image are designated using the touch panel 5, the distanceinformation for these two or more points are read out from the internalmemory 2, the tilt direction and tilt angle of the object plane withrespect to the optical axis of the lens are individually calculated, andthe object plane is set based on the tilt direction and tilt angle,under control of the controller 7. A method for calculating the tiltdirection and tilt angle is described in detail later.

In this way, the object plane perpendicular to the optical axis of theimaging lens, or one point not to be blurred, is designated using, e.g.,the touch panel 5, and predetermined image processing such as blurringis performed based on the distances from the object plane. This providesan image given an effect equal to that of “imaging with such a shallowdepth of field and tilted object plane” that imaging may not beperformed by adjusting a focal length by adjusting the performance of anactual imaging device, lens position, and the like, without usingoptical camera movement effects (tilt movement or swing movement). Thus,when the image processing apparatus in accordance with the firstembodiment is applied to an imaging apparatus, an image as if obtainedusing advanced techniques with a digital single lens reflex camera orlarge format camera can be easily obtained.

Next, an image processing apparatus in accordance with the firstembodiment of the invention employing the above-described basicprinciple is described in detail with reference to a block diagram shownin FIG. 3. In this figure, the image processing apparatus is exemplarilyapplied to an imaging apparatus such as a digital camera, but is notlimited thereto.

As shown in FIG. 3, an imaging device 12 is placed on the optical axisof an imaging optical system 11 and is connected to an image outputsection 13. The image output section 13 is connected to ananalog-to-digital (A/D) converter 14. The A/D converter 14 is connectedto a memory controller 15 and an image processor 24. The memorycontroller 15 is communicatably connected to a display memory 16. Thememory controller 15 is further connected to an LCD monitor 19 via adisplay processor 17 and a digital-to-analog (D/A) converter 18. Theimage processor 24 is connected to a frame buffer (frame memory) 25including DRAM (dynamic random access memory) and the like.

A CPU (central processing unit) 21 is responsible for controlling awhole apparatus, and is communicatably connected to a ROM (read onlymemory) 22, a RAM (random access memory) 23, the image processor 24, anOSD (on-screen display) processor 26, an imaging condition detector 27,a camera drive controller 28, a range finder 29, and a touch panel 34via a control bus 35.

Furthermore, the CPU 21 is connected to a memory card 31 via the controlbus 35 and a memory interface (I/F) 30, and is connected to anoperational input section 33 via the control bus 35 and an input I/F 32.In this configuration, the CPU 21, the ROM 22, and the RAM 23 form acontroller 20.

In this configuration, the RAM 23 and the memory card 31 provide astorage capability, for example. The operational input section 33provides a designation capability and a setting capability, for example.The image processor 24 provides an image processing capability, forexample. The controller 20, more specifically the CPU 21, provides acontrol capability, for example.

In this configuration, the CPU 21 controls the components of theapparatus and reads out an image processing program from the ROM 22 toperform processing according to the program. The ROM 22 stores variousfixed data in addition to the image processing program. In theprocessing, the RAM 23 serves as a work area.

An incident light from an object is imaged on the imaging device 12through the imaging optical system 11. The camera drive controller 28performs various drive controls such as exposure control, aperturecontrol, focus control, zoom control, and shutter control. The imagingdevice 12 performs photoelectric conversion and provides an analog imagesignal to the image output section 13. The image output section 13provides the analog image signal to the imaging condition detector 27.The imaging condition detector 27 detects the imaging conditionincluding white balance for the analog image signal.

The image output section 13 further sends the analog image signalincluding a luminance signal and a color difference signal to the A/Dconverter 14. The A/D converter 14 converts the analog image signal to adigital image data and sends the image data to the memory controller 15.The memory controller 15 temporarily stores the image data in thedisplay memory 16. The image data is read out through the memorycontroller 15 and sent to the display processor 17. The displayprocessor 17 converts the image data to an RGB image data. The RGB imagedata is converted to an RGB analog image signal for liquid crystaldriving by the D/A converter 18 and sent to the LCD monitor 19. Then theLCD monitor 19 displays alive image. The live image is referred to asthrough image.

With the live image displayed on the LCD monitor 19 in this way, when ashutter button, which is one element of the operational input section33, is pressed, an imaging operation is performed.

Specifically, under the control by the controller 20, one frame of imagedata of the live image is captured into the frame buffer 25 through theimage processor 24, compressed by the image processor 24, converted toan image file in a predetermined format, and recorded to the memory card31 through the memory I/F 30. At the same time, information on thedistance to the object measured by the range finder 29 including theimaging device (abbreviated to “distance information” hereinafter) isalso recorded to the memory card 31 in a given format.

When the controller 20 reads out the image data and the distanceinformation from the memory card 31, the compressed image data isdecompressed by the image processor 24, stored in the frame buffer 25,and at the same time, written to the display memory 16 by the memorycontroller 15, read out from the display memory 16 to display the imageon the LCD monitor 19.

In this state, when the user designates a desired point on the touchpanel 34, the CPU 21 temporarily sets the point in the captured image asdesignated point. At this point, an object plane is temporarily setbased on the set point. Furthermore, when a cross-shaped key, which isone element of the operational input section 33, is operated, the CPU 21sets the tilt direction and tilt angle of the object plane based on thatoperation. In this way, on the image data stored in the frame memory 25,the CPU 21 of the controller 20 performs image processing using theimage processor 24.

More specifically, under the control by the CPU 21, the image processor24 performs image processing to blur pixels that are not on the objectplane so that the same result can be obtained as a photograph taken witha film camera in which focus is achieved on shallow depth of field andthe rest is blurred. The image data on which this blurring was performedis sent to the display processor 17 through the memory controller 15 todisplay the processed image on the LCD monitor 19. The processed imagedata is recorded to the memory card 31 through the memory I/F 30.

For blurring, the image processor 24 can also control the blur amountdepending on the distance from the object plane. By this control, theblur amount can be gradually changed depending on the distance betweenthe object plane and pixels. For example, increasing the blur amount asthe distance between the object plane and the other pixels increases canprovide blurring effect similar to that of the imaging optical system.Specifically, this can provide an effect such that the object plane isfocused and the blur amount increases (and the image becomesincreasingly blurred) as the distance from the object plane increases.In this example, the blur amount is controlled based on the relation ofthe distance between the object plane and the other pixels. However,similar control may be performed for each given region including two ormore pixels, rather than for each pixel.

Next, a characteristic process performed by the image processingapparatus in accordance with the first embodiment of the invention toprovide a correction equal to camera movement effect is described indetail below with reference to flowcharts shown in FIGS. 4 to 6.

The process described below also corresponds to the image processingmethod in accordance with the first embodiment.

Starting the process shown in FIG. 4, the CPU 21 of the controller 20performs imaging (step S1), generates distance information (step S2),and records image data and the distance information to the memory card31 (step S3). Of course, performing imaging and generating distanceinformation may be performed in any order or at the same time. Therecording destination is not limited to the memory card 31.

Specifically, through the above steps, under the control by the CPU 21of the controller 20, image data is captured into the frame memory 25through the image processor 24, compressed by the image processor 24,converted to an image file in a predetermined format, and recorded tothe memory card 31 through the memory I/F 30. At the same time, distanceinformation measured by the range finder 29 is also recorded to thememory card 31 in a given format.

In this example, imaging is performed by one imaging device 12, anddistance information is generated by an imaging device (not shown) thatis a component of the range finder 29. However, instead of using thesetwo imaging devices, one imaging device 12 may be shared for performingimaging and generating distance information. Of course, performingimaging and generating distance information may be parallel processingor serial processing.

Methods for obtaining distance information include TOF method and depthfrom defocus method. “TOF” method is a technique for determining thedistance to an object by measuring propagation delay time of a lightduring which the light is emitted from a light emitting device includedin the range finder 29, is reflected by the object, and returns. “Depthfrom defocus” method is a technique for determining distance based onthe result of analyzing blur amount in an image. When two imagingdevices are used, the blur amount for imaging and the blur amount forranging can be separately optimized by changing at least one ofcharacteristic, size, and aperture condition between the imagingdevices. When only one imaging device 12 is used, the blur amount forimaging and the blur amount for ranging can be optimized by changing,for example, aperture condition for between imaging and ranging.However, the means are not limited to these.

The captured image thus recorded is displayed on the LCD monitor 19(step S4).

One example of the display is as shown in FIG. 7(a). In this example,objects O₁ to O₄ are shown. The distance information for the capturedimage is as shown in FIG. 7(b). This figure shows that low-key objectsare in near field and high-key objects are in far field. In thisexample, the distance information is what is called depth map data,which is the individual distance data for one or more points in thecaptured image. The distance data for each pixel can be obtained at themaximum.

Then a designated point is temporarily set in the captured image (stepS5). The detail of the process in this subroutine is, for example, asshown in the flowchart of FIG. 5.

Specifically, when a user designates an desired designated point in thecaptured image by operating the touch panel 34 (step S21), the CPU 21determines whether it is a first designated point or not (step S22). Ifthe CPU 21 determines that the designated point that has just beendesignated is the first one (Yes in step S22), the CPU 21 temporarilysets a plane containing the first designated point designated andperpendicular to the optical axis as a first object plane (step S23),stores information related to the first object plane (coordinates of thefirst designated point) in the RAM 23 (step S27), and displays the firstdesignated point designated, on the captured image displayed on the LCDmonitor 19 (step S28).

Then the CPU 21 determines whether or not the user has requested byoperating the operational input section 33 and the like to finishdesignating designated points (step S29). If the CPU 21 determines thatanother designated point will be designated without finishing (Yes instep S29), the process returns to step S21 to repeat the above-describedsteps. If the CPU 21 determines that the designation of designatedpoints should be finished (No in step S29), the process ends andreturns.

On the other hand, in step S22, if the CPU 21 determines that thedesignated point that has just been designated is not the first one (Noin step S22), the CPU 21 determines whether that designated point is asecond one or not (step 24).

If the CPU 21 determines that the designated point that has just beendesignated is the second one (Yes in step S24), the CPU 21 performsconversion of the first object plane by rotating the first object planeon the X-axis and on the Y-axis, and temporarily sets the convertedplane that further contains the second designated point as a secondobject plane (step S25). Then the CPU 21 stores information on thesecond object plane (coordinates, tilt angle, and tilt direction of thefirst and second designated points) in the RAM 23 (step S27), anddisplays the first and second designated points designated, on thecaptured image displayed on the LCD monitor 19 (step S28). However, onlytwo points are not sufficient information for temporarily setting theobject plane, because two or more planes may contain the line connectingthe two points. Then the CPU 21 may temporarily rotate the first objectplane on the Y-axis (or the X-axis) and set the rotated plane containingthe second designated point as a second object plane.

Then, again, the CPU 21 determines whether or not the user has requestedby operating the touch panel 34 and the like to designate anotherdesignated point (step S29). If the CPU 21 determines that anotherdesignated point will be designated (Yes in step S29), the processreturns to step S21 to repeat the above-described steps. If the CPU 21determines that another designated point will not be designated (No instep S29), the process ends and returns.

On the other hand, in step S24, if the CPU 21 determines that thedesignated point that has just been designated is not the second one (Noin step S24), the CPU 21 determines that the designated point is a thirdone. Then the CPU 21 rotates the second object plane on the X-axis andon the Y-axis, and temporarily sets the rotated plane that furthercontains the first to third designated points as a third object plane(step S26). Furthermore, the CPU 21 stores information on the thirdobject plane (coordinates, tilt angle, and tilt direction of the firstto third designated points) in the RAM 23 (step S27), and displays thefirst to third designated points designated as the above, on thecaptured image displayed on the LCD monitor 19 (step S28).

Then, again, the CPU 21 determines whether or not the user has requestedby operating the touch panel 34 and the like to designate anotherdesignated point (step S29). In this example, with the assumption thatthe number of designated points that can be designated is limited to upto three, designation of another designated point is not allowed (No instep S29), and the process returns to the main routine (from step S6 ofFIG. 4). This limitation of up to three designated points is only forconvenience of description of the example. However, an extension fordesignating four or more designated points is possible, of course.

When the first to third designated points are designated through theabove-described process, the conversion of distance information of theobject plane determined by those designated points is performed. Forexample, when three designated points are designated, the distanceinformation is converted to depth map data related to the distanceinformation of the three designated points as shown in FIG. 7(c).

Returning to FIG. 4, the CPU 21 performs the conversion of the distancedata of each point of the image data based on the information, e.g., thecoordinates, tilt angle, and tilt direction, of the set object plane(any of the first to third object planes), and performs an imageprocessing using the image processor 24 based on the converted distancedata (step S6). Specifically, for example, blurring is performed onpoints on the object plane so as to obtain an image that is given aneffect equal to that of “imaging with such a shallow depth of field andtilted object plane” that imaging may not be performed with an actualimaging device 12 or focal length. Then the CPU 21 displays the blurredimage on the LCD monitor 19 (step S7).

Now, the conversion of the distance data based on the setting of theobject plane is described with reference to FIG. 8.

In FIG. 8, the direction of the X-axis is perpendicular to the opticalaxis. When a first designated point P₁ is designated, a plain containingthe first designated point P₁ and perpendicular to the optical axis areset as a first object plane. With this setting, the first designatedpoint P₁ becomes the center of a new coordinate system, with thecoordinates converted from (X₁, Y₁, Z₁) to (0, 0, 0). Along with thiscoordinate system conversion, the coordinates of a point P_(n) on thefirst object plane is converted, for example, from (X_(n), Y_(n), Z_(n))to (X_(n)-X₁, Y_(n)-Y₁, Z_(n)-Z₁). This converted coordinates arerepresented as (X_(n)′, Y_(n)′, Z_(n)′).

When a second designated point P₂ is additionally designated, a secondobject plane containing the first designated point P₁ and the seconddesignated point P₂ is set. The first designated point P₁ is still thecenter of another new coordinate system, with the coordinates of (0, 0,0). The tilt angle of the second object plane with respect to the firstobject plane is θ. Assuming that the tilt direction is the Z-axis(positive) direction, for example, the point P_(n) on the first objectplane is projectively transformed to information on the distance fromthe second object plane with the Z-coordinate converted toZ_(n)′−X_(n)′·tan θ. This is the distance information on the point P_(n)after conversion. Here, for convenience of description, the conversionof the point P_(n) on the first object plane is described. However,similar conversion of distance information is performed on every point(for example, pixel by pixel). The same applies to the first objectplane rotated on the Y-axis with the tilt angle of θ.

Additionally, when a third designated point is designated and a thirdobject plane is set (not shown in FIG. 8), the distance information isconverted additionally based on the tilt angle and tilt directionthereof. Assuming that the third object plane is tilted with respect tothe first object plane with the tilt angle of α on the X-axis, theZ-coordinate of the point P_(n) mentioned above is Z_(n)′−X_(n)′·tanθ−Y_(n)′·tan α. This is the distance information on the point P_(n) withthe respect to the third object plane. The same applies to the firstobject plane rotated on the Y-axis with the tilt angle of θ andadditionally rotated on the X-axis with the tilt angle of α.

Specifically, for example, when the first to third designated points aredesignated on the objects O₁ to O₃, respectively, or when the firstdesignated point is designated, and through a rotation of the objectplane on the X- and/or Y-axis, the objects O₁ to O₃ are set to be on theobject plane, blurring is performed according to the distance from theobject plane, as shown in FIG. 7(d). In other words, the image data isblurred with the blur amount according to the converted distanceinformation. More specifically, the image data is blurred such that theblur amount increases as the distance from the object plane increases.Thus, in the example shown in FIG. 7(d), the object O₄ farthest from theobject plane is most blurred. It should be noted that the image data maybe blurred so that points within a predetermined distance from theobject plane will be blurred with the same blur amount. In this way,depth of field can be considered along with blurring.

Next, the CPU 21 determines whether or not the user has requested byoperating the touch panel 34 and the like to adjust the object plane(step S8). If the CPU 21 determines that the adjustment is not to beperformed (No in step S8), the image data of the image blurred asdescribed above is recorded to the memory card 31 (step S12) and thewhole process ends. In step S12, the image data before being blurred mayalso be recorded to the memory card 31. It is recommended that this datashould be managed with a different number by incrementing the lastnumber. On the other hand, the CPU 21 determines that the adjustment isto be performed (Yes in step S8), the tilt direction and tilt angle ofthe object plane with respect to the optical axis of the lens is set.

The detail of the setting is as shown in the flowchart of FIG. 6.

Specifically, jumping to the process shown in FIG. 6, first, the CPU 21determines whether or not a cross-shaped key, which is one element ofthe operational input section 33, has been operated (step S41). If theCPU 21 determines that it has not been operated (No in step S41), theprocess ends and returns.

On the other hand, if the CPU 21 determines that the cross-shaped keyhas been operated, then the CPU 21 determines whether the operated keyis the Up key or not (step S42). If the CPU 21 determines that theoperated key is the Up key (Yes in step S42), the CPU 21 rotates theobject plane on the X-axis with the upper portion tilted toward the backand proceeds to step S50 (step 43).

On the other hand, if the CPU 21 determines that the operated key is notthe Up key (No in step S42), then the CPU 21 determines whether theoperated key is the Down key or not (step S44). If the CPU 21 determinesthat the operated key is the Down key (Yes in step S44), the CPU 21rotates the object plane on the X-axis with the lower portion tiltedtoward the back and proceeds to step S50 (step 45).

On the other hand, if the CPU 21 determines that the operated key is notthe Down key (No in step S44), then the CPU 21 determines whether theoperated key is the Right key or not (step S46). If the CPU 21determines that the operated key is the Right key (Yes in step S46), theCPU 21 rotates the object plane on the Y-axis with the right portiontilted toward the back and proceeds to step S50 (step S47).

On the other hand, if the CPU 21 determines that the operated key is notthe Right key (No in step S46), then the CPU 21 determines whether theoperated key is the Left key or not (step S48). If the CPU 21 determinesthat the operated key is the Left key (Yes in step S48), the CPU 21rotates the object plane on the X-axis with the left portion tiltedtoward the back and proceeds to step S50 (step S49).

Then if the CPU 21 determines that the operated key is not the Left key(No in step S48), the CPU 21 determines that the operation has beenwrong and proceeds to step S50.

Then the CPU 21 determines whether the setting is to be ended or not,according to the operational input using the enter key, which is oneelement of the operational input section 33 (step S50). If the CPU 21determines that the setting is not to be ended, the process returns tostep S41 to repeat the above-described steps. On the other hand, if theCPU 21 determines that the setting is to be ended, the CPU 21 stores thesetting information on the set object plane in the RAM 23 (step S51),and the process returns to the main routine (from step S10 of FIG. 4).

The above-described process shown in FIG. 6 allows the object plane tobe adjusted to a desired tilt direction and tilt angle by rotating theobject plane on the X-axis or Y-axis based on the operation of thecross-shaped key (Up, Down, Left, and Right keys). For example,incremental/decremental adjustment for stepwise changing tilt angle by apredetermined angle for each operation of the cross-shaped key is alsopossible. This allows for precise adjustment in order to put an intendedpoint on the object plane, or in order to obtain an image that is givenan effect equal to that of imaging with as shallow depth of field asthat of camera movement effect for an intended point.

Returning to FIG. 4, the CPU 21 performs image processing based on theset object plane and the distance information (step S10), and displaysthe image on which the image processing was performed on the LCD monitor19 (step S11). The detail of the process in steps S10 and S11 is similarto that of steps S6 and S7 as already described. So the description isnot repeated here.

In this way, the process returns to step S8, and the CPU 21 determineswhether the adjustment of the object plane is to be performed again ornot. If the CPU 21 determines that the adjustment is to be performedagain, the CPU 21 repeats the process from step S9. If the CPU 21determines that the adjustment is not to be performed again (No in stepS8), the image data of the image blurred as described above is recordedto the memory card 31 (step S12) and the whole process ends.

Now, as a method for obtaining distance information, TOF method isdescribed in more detail with reference to a conceptual drawing shown inFIG. 9. As shown in FIG. 9, the range finder 29 includes a lightemitting device 29a, an imaging lens 29b, and an imaging device 29c.With this configuration, a light with a certain wavelength modulated isemitted from the light emitting device 29a and irradiated to an object200. The light reflected by the object 200 is received by the imagingdevice 20c through the imaging lens 29b. Then, by determining “delaytime” for each pixel of the imaging device 20c from the phase differencebetween the irradiated light and the reflected light, the distance d tothe object, which is distance information, is obtained based on thedelay time. For example, this distance information may be found bydividing the product of light speed and the delay time by 2. Through theabove-described process, the embodiment can provide distance informationfor each pixel at the maximum. In order to obtain distance information,phase difference analysis may also be used.

As described above, according to the first embodiment of the invention,an image given an effect corresponding to “camera movement (swing,tilt)” can be obtained with simple operation by obtaining distanceinformation corresponding to an image imaged through an imaging opticalsystem, designating a point in the captured image to set an objectplane, and performing image processing based on the distance informationand information on the object plane.

Of course, this capability can be implemented in an imaging apparatussuch as a low-cost and compact digital still camera.

Specifically, according to an embodiment of the invention, an imagegiven the effect corresponding to “camera movement (swing, tilt)” can beeasily obtained, thus, for example, eliminating the need for acomplicated mechanism for enabling optical camera movement such as swingand tilt (a mechanism design according to what is called “Scheimpflugprinciple”), and eliminating the need for an optical design specializedfor “camera movement (swing, tilt)”. Then a low-cost and compact imageprocessing apparatus and image processing method can be provided.

It should be noted that a generated object plane may be not only aplane, but may be any curved surface (e.g., variable field curvature).Specifically, for example, when a plane perpendicular to the opticalaxis is set as an object plane as shown in FIG. 10(a), then a curvedsurface formed by bending the left and right portions of the objectplane toward an image synthesizer with a predetermined curvature can beset as an object plane as shown in FIG. 10(b), or a curved surfaceformed by bending the left and right portions of the object plane towardopposite side of the image synthesizer with a predetermined curvaturecan be set as an object plane as shown in FIG. 10(c). Of course, thoughnot shown, a curved surface formed by bending the upper and lowerportions of the object plane toward the image synthesizer (or towardopposite side of the image synthesizer) with a predetermined curvaturecan be set as an object plane. Also, an image plane can be inmortar-shape. This will be discussed in detail in a ninth embodiment ofthe invention.

Furthermore, as shown in FIG. 11, how the object plane is being adjustedusing a cross-shaped key, which is one element of the operational inputsection 33, may be displayed on a window 351 located at the lower-rightof a screen 350 of the LCD monitor 19. This will be discussed in detailin an eighth embodiment of the invention.

(Second Embodiment)

Next, a second embodiment of the invention is described.

The second embodiment of the invention implements the setting of anobject plane and blurring using an image processing apparatus asdescribed previously with an information processing apparatus such as acomputer and accessory hardware.

FIG. 12 shows an example of configuration of an information processingapparatus in accordance with the second embodiment.

As shown in FIG. 12, an information processing apparatus 50 includes aninput section 51, a controller 52 for controlling a whole system basedon a control program 53, a display 54, an operational input section 55,and a storage 56. Based on the control program 53, the controller 52controls, for example, an object plane setting capability 53a, a tiltsetting capability 53b, and an image processing capability 53c. Thiscontrol program corresponds to, for example, an image processingprogram.

In this configuration, when image data and distance information is inputfrom the input section 51, the controller 52 reads out the controlprogram 53 stored in the storage 56 and performs processing based on thecontrol program 53. Specifically, the controller 52 displays an imagebased on the image data on the display 54, and prompts a user to set anobject plane by operating the operational input section 55.

When the user selects a desired point (or region) in the displayed imageby operating the operational input section 55, the controller 52 sets anobject plane based on the object plane setting capability 53a. Then thecontroller 52 sets the tilt direction and tilt angle of the object planebased on a further operation of the operational input section 55. Thenthe controller 52 performs image processing such as blurring based onthe information on the finally set object plane and distanceinformation, and displays the processed image on the display 54. Thedetail of the setting of the above example is similar to that of thefirst embodiment (see the flowcharts shown in FIGS. 4 to 6). So thedescription is not repeated here.

As described above, according to the second embodiment of the invention,an image given the effect corresponding to “camera movement (swing,tilt)” can be obtained with simple operation by prompting the user toset an object plane based on the input image data and distanceinformation and performing an image processing based on the informationon the object plane and the distance information.

In other words, the second embodiment of the invention can provide animage processing program for easily obtaining an image given the effectcorresponding to “camera movement (swing, tilt)”. Also, by implementingthis image processing program, an image given such an effect can beobtained easily and at a low cost using an information processingapparatus such as a computer.

In the above, the first and second embodiments of the invention has beendescribed. However, of course, the invention is not limited to thoseembodiments, but various improvements or modifications may be madewithout departing from the spirit of the invention.

For example, in the above-described first and second embodiments, up tothree desired point are designated using the touch panel while lookingat the screen and an object plane is set based on the point. However theinvention is not limited to these examples. The user may designate threepoints and also specify a depth of field (aperture) to adjust variationof blurring (using aperture priority provides an effect equal to that ofimaging with such a shallow depth of field and tilted object plane thatimaging may not be performed with an actual imaging device/focallength). The depth of field (aperture) may be fixed automatically.Furthermore, the user may designate four or more points and maycalculate the depth of field so as to focus on all of the points or givesome weight to each of the points. “Depth of field” means a range ofdistance within which focus is considered to be achieved around anobject in focus.

The user may specify a focal length and aperture by 35 mm sizeconversion. Here, 35 mm size conversion means considering a lens ofwhich angle of view is equal to that of 35 mm standard lens for eachdifferent imaging size as a standard lens. Or when four or more pointsare designated, an object plane can be determined based on ordinarysolid analytic geometry. The same applies to the conversion of distanceinformation based on information on an object plane. Of course, blurringalgorithm is not limited to the above-described example.

The image processing is not limited to the above-described blurring, butincludes various processing such as granulation, mosaic, changingtexture, noise addition, changing luminance or chrominance, and stepwisegradation. Of course, an object plane may be a curved surface, such asspherical surface, cylindrical surface, conical surface, andhigher-order approximated curved surface.

(Third Embodiment)

Next, a third embodiment of the invention is described.

FIG. 13 shows a configuration of an image processing apparatus inaccordance with the third embodiment.

In this figure, the image processing apparatus in accordance with thethird embodiment is exemplarily applied to an imaging apparatus such asa digital camera.

As shown in FIG. 13, the image processing apparatus includes a camera60, a camera DSP (digital signal processor) 61, an SDRAM (synchronousdynamic random access memory) 63, a media I/F (media interface) 64, acontroller 62, an operation section 66, an LCD (liquid crystal display)controller 67, an LCD 68, an external I/F (external interface) 69, and arecording media 65. The recording media 65 is detachable.

The recording media 65 may be what is called a memory card employingsemiconductor memory, an optical recording media such as recordable DVD(digital versatile disk) and recordable CD (compact disc), a magneticdisk, and the like.

The camera 60 includes an optical system block 60a, an aperture 60b, animaging device 60c such as CCD (charge coupled device), an A/D converter60d, an optical block driver 60e, an aperture driver 60f, an imagingdevice driver 60g, a timing generator 60h, a drive section 60i, adetector 60j, and a blur detector 60k. The optical system block 60aincludes a lens, a focusing mechanism, a shutter mechanism, an aperture(iris) mechanism, and a hand blur correction driver. For the lens in theoptical system block 60a, a zoom lens is used.

The controller 62 is an microcomputer in which a CPU (central processingunit) 62a, a RAM (random access memory) 62b, a flash ROM (read onlymemory) 62c, a clock circuit 62d, and the like are connected via asystem bus 62e, and is responsible for controlling components of theimaging apparatus in accordance with this embodiment.

The RAM 62b is mainly used as a work space, for example, for temporarilystoring an intermediate result of processing. The flash ROM 62c stores,in advance, various programs to be executed by the CPU 62a and data tobe used in processing. An image processing program may be stored in theflash ROM 62c. The clock circuit 62d can provide the current date, thecurrent day of the week, and the current time of the day, and can alsoprovide a shooting date and time.

In capturing an image, the optical block driver 60e generates a drivesignal, in response to the control from the controller 62, for actuatingthe optical system block 60a and provides this signal to the opticalsystem block 60a to actuate the optical system block 60a. In response tothe drive signal from the optical block driver 60e, the optical systemblock 60a captures an image of an object and provides it to the imagingdevice 60c.

The imaging device 60c, which is responsible for performingphotoelectric conversion on the image from the optical system block 60a,operates in response to the drive signal from the imaging device driver60g and captures the image of the object from the optical system block60a. Furthermore, based on a timing signal from the timing generator 60hcontrolled by the controller 62, the imaging device 60c provides ananalog image data of the captured image of the object to the A/Dconverter 60d. The A/D converter 60d performs an A/D conversion andgenerates a digitized image data.

This digitized image data is provided to the camera DSP 61. The cameraDSP 61 performs camera signal processing, such as AF (auto focus), AE(auto exposure), and AWB (auto white balance), on the provided imagedata. The image data on which various adjustments have been performed inthis way is compressed with a predetermined compression scheme, providedto the recording media 65 through the system bus 62e and the media I/F64, and then recorded to the recording media 65.

After the image data is recorded in the recording media 65, a desiredimage data is read out from the recording media 65 through the media I/F64 in response to an operational input from the user accepted throughthe operation section 66 including a touch panel and control keys, andprovided to the camera DSP 61.

The camera DSP 61 decompresses (expands) the compressed image data readout from the recording media 65 and provided through the media I/F 64,and provides the decompressed image data to the LCD controller 67through the system bus 62e. The LCD controller 67 generates an imagesignal for the LCD 68 from the provided image data and provides theimage signal to the LCD 68. This enables an image corresponding to theimage data recorded in the recording media 65 to be displayed on thedisplay screen of the LCD 68.

Also, image data provided through the external I/F 69 (for example,image data provided from an external personal computer connected withthe external I/F 69) can be recorded to the recording media 65. Theimage data recorded in the recording media 65 can be provided to anexternal personal computer and the like through the external I/F 69.

Furthermore, with the external I/F 69 connected to a communicationmodule to connect to a network such as the Internet, various image dataand other information can be obtained through the network and recordedto the recording media 65. Or the data recorded in the recording media65 can be sent to a destination through the network.

The external I/F 69 may be a wired interface such as IEEE (institute ofelectrical and electronics engineers) 1394 and USB (universal serialbus). Or the external I/F 69 may be a wireless interface using light orradiowave. Thus the external I/F 69 may be wired or wireless.

As described above, the image processing apparatus in accordance withthe third embodiment is directed to provide a more simple imagingtechnique specifically corresponding to “camera movement (swing, tilt)”,and provide this using an image processing and image correctioncontrolled by the CPU 62a, and additionally allows intuitive operationas one feature.

FIG. 14 shows an appearance configuration of a digital camera as animaging apparatus to which the image processing apparatus is applied.FIG. 14(a) is a front perspective view of the digital camera. FIG. 14(b)is a rear perspective view of the digital camera. As shown in thesefigures, a digital camera 70 includes a lens 71, a shutter button 72, amenu button 73, an enter button 74, an exposure compensation button 75,a touch panel 76, and a cross-shaped key 77. Designation of a point(described later) is performed by operating the touch panel 76 and thelike.

Now, a characteristic process using the image processing apparatus inaccordance with the third embodiment of the invention is described indetail with reference to a flowchart shown in FIG. 15. Some or all ofthe process described below also corresponds to the image processingmethod in accordance with the third embodiment of the invention.

Starting this process, the CPU 62a of the controller 62 performs imagingon the object using the imaging device 60c and generates a digital imagedata using the A/D converter 60d (step S101). Then the CPU 62a generatesdistance information (depth map) for the generated image data (stepS102). The distance information (depth map) is distances to the objecteach related to each pixel of the image, and can be obtained using TOF(time of flight) method or depth from defocus method.

Then the CPU 62a records the image data and the distance information tothe recording media 65 (step S103). In other words, the image data andthe distance information are recorded to the recording media 65 throughthe media I/F 64 in association with each other. The distanceinformation is maintained in the form of metadata. Storing the imagedata and the distance information associated with each other using afile number and the like to different locations may reduce the size ofimage files themselves. In this case, the association between themshould be determined when using the image data and the distanceinformation.

Then the CPU 62a controls the LCD controller 67 to display the image onthe LCD 68 (step S104). Then the CPU 62a performs processing based onthe manipulation of the object plane corresponding to “camera movement(swing, tilt)” (step S105). The detail of the process of step S105 isdescribed later with reference to FIG. 16.

The object plane is a plane that intersects with the extendedlight-accepting surface of the imaging device 60c and the extendedprincipal plane of the optical system and on which focus is achieved,conforming to what is called Scheimpflug principle. In the imageprocessing apparatus accordance with this embodiment, imaging isperformed with the optical axis of the lens (almost) perpendicular tothe imaging device plane, so that what is called camera movement (swing,tilt) operation cannot be performed using the imaging device 60c and theoptical system of the optical system block 60a. The manipulation of theobject plane corresponding to “camera movement (swing, tilt)” isperformed using image processing.

Then the CPU 62a reads out the image data recorded in step S103 andperforms image processing (blurring according to the distance from theobject plane) using the object plane set in step S105 (step S106). Thedetail of the process of step S106 is described later with reference toFIGS. 18 and 19. Then the CPU 62a records the image data on which theimage processing has been performed to the recording media 65 throughthe media I/F 64 (step S107).

At this point, if the CPU 62a, when performing imaging, recorded imagedata for the displayed image to the recording media 65 as a differentfile than the original image data, the CPU 62a records at the same timeimage data for the displayed image on which the image processing hasbeen performed. Then the process ends.

Next, the detail of the process of step S105 in FIG. 15 is describedwith reference to a flowchart shown in FIG. 16. In this process, thetouch panel 76 is used for the manipulation of the object plane.

Entering this process, the CPU 62a accepts the designation of the centerof the object plane by touching the touch panel 76 (step S201).Specifically, the CPU 62a detects the coordinates of a touched locationand accepts the location as a designation of location. The center of theobject plane is the rotation center of the object plane and is thethree-dimensional and spatial center location on which the object planerotates in a new coordinate system consisting of X-, Y-, and Z-axes.

Then the CPU 62a displays a point representing the rotation center ofthe object plane on the display screen of the LCD 68 to let the userknow the rotation center of the object plane thus set (step S202). Thenthe CPU 62a performs coordinate conversion on the distance information(depth map) so that the rotation center of the object plane will be thecenter of the three-dimensional coordinate system (step S203).

The coordinates (X, Y, Z) of an image is defined by the two-dimensionalcoordinates (X, Y) and the depth coordinate (Z) of the image. Thedistance information (depth map) relates to the depth coordinate (Z) ofthe coordinates (X, Y, Z). The CPU 62a first performs conversion so thatthe coordinates (X, Y, Z) of the pixel set to the center of the objectplane will be the origin (0, 0, 0) of the three-dimensional coordinatesystem. The coordinates of the other points (pixels) are converted tocoordinates of a new coordinate system of which center is thecoordinates of the center of the object plane. In this way, the distanceinformation is converted to the distance from the object plane. Thedetail of this conversion of the distance information is as previouslydescribed with reference to FIG. 8. So the description is not repeatedhere.

In this embodiment, the shape of the object plane is assumed to be asimple plane. When the center of the object plane on the display isshifted from the user-intended position, the correction is made bydragging the displayed point. For example, the center point can be movedin the direction specified by using the cross-shaped key 77 shown inFIG. 14(b).

Then the CPU 62a determines whether the user has fixed the center of theobject plane by operating the touch panel 76 (e.g. tapping the enterbutton) (step S204). When the center of the object plane has beencorrected according to the user operation, the coordinates of the centerchanges. In this case, the process returns to step S203, and thecoordinate conversion is performed again on the distance information(depth map) so that the corrected center of the object plane will be thecenter of the three-dimensional coordinate system. On the other hand,the center is fixed by, for example, tapping again the center of theobject plane displayed on the LCD 68, the process proceeds to step S205.

If the CPU 62a determines that the center of the object plane is fixedby the user operation, the CPU 62a performs image processing on thedisplayed image and displays the image so that the effect will be seenwhen the object plane is manipulated (step S205). This displayed imageis not the captured image data itself, but is an image written to theRAM 62b (or SDRAM 63) for displaying on the display screen (LCD 68) forthe purpose of confirming the image itself or the effect of manipulatingthe object plane. This image processing may be a simplified processingbecause the purpose is only to confirm the image itself.

This displayed image may be downsized from the captured image data to animage matched with the resolution of the display screen (LCD 68) andtemporarily stored in the RAM 62b (or SDRAM 63). Or, when imaging isperformed, the displayed image may be stored in the recording media 65through the media I/F 64 as a different file than the original imagedata.

Specifically, in step S205, blurring is performed according to thedistance from the object plane in response to the distance from theobject plane. The size of the displayed image may be smaller than thatof the actually recorded image. It is desirable to reduce processingload and enable smooth user operation by displaying an image with aminimum size for the size and number of pixels of the display screen.

In this embodiment, it is assumed that only the object plane ismanipulated with a fixed depth of field (however, shallower than that ofactual imaging) for simple user operation.

Then the CPU 62a rotates the object plane according to the useroperation (step S206). This rotation of the object plane is performed byrotating (or tilting) the object plane on the X- or Y-axis by tappingcoordinates other than the center of the object plane.

More specifically, as shown in FIG. 17(a), the direction of tilting theobject plane is specified by a tap (touch) location 500a on a displayscreen 500, and the amount of tilting the object plane is specified bytapping intensity. In the display screen 500, the object plane isrotated on a rotation axis 500c that is perpendicular to the lineconnecting the coordinates (X, Y) of the tap location 500a and arotation center 500b related to the designated point previouslydesignated on the object plane, and passes through the rotation center500b of the object plane. Tapping gives an intuitive feeling of hittinga plane.

The rotation of the object plane is performed by a coordinate conversionbased on solid analytic geometry. For example, the distance from theobject plane is calculated for each pixel. Specifically, according tothe user operation of designating the rotation center of the objectplane and rotating the object plane, the CPU 62a performs coordinateconversion from the distance information to the information on thedistance from the object plane.

Then the CPU 62a performs image processing on the displayed image sothat the effect of thus rotating the object plane on its rotation center500b (or tilting the object plane) can be seen, and displays thedisplayed image on the LCD 68 (step S207). In other words, the CPU 62aperforms blurring according to the distance from the rotated (tilted)object plane. The result is shown in FIG. 17(b).

As with the image processing in step S205 described above, the size ofthe displayed image on which the image processing in step S207 isperformed may be minimum for both smooth user operation and confirmingthe effect of the operation. Even after tilting the object plane, therotation center of the object plane may be corrected by dragging. Thismay be achieved by performing coordinate conversion as is with step S203described above.

Then the CPU 62a determines whether or not the user has requested by,for example, tapping again the center of the object plane displayed onthe touch panel 76 to finish rotating the object plane (step S208). Ifthe CPU 62a determines that the user has not requested to finishrotating, the process returns to step S206 to repeat the above-describedsteps. On the other hand, if CPU 62a determines that the user hasrequested to finish rotating, the process ends and returns to theprocess shown in FIG. 15.

Next, the detail of the process of step S106 in FIG. 15 is describedwith reference to a flowchart shown in FIG. 18. Here, it is assumed thatthe image processing apparatus has a table based on the relation asshown in FIG. 19 in the flash ROM 62c and the like. In FIG. 19, thevertical axis represents the value of the blurring parameter and thehorizontal axis represents the distance information. The blurringparameter is zero when the distance is less than a depth of field. Whenthe distance is equal to or more than the depth of field, the blurringparameter is, for example, a linear function of the distance (that is,the blurring parameter increases as the distance increases). It ispreferred that some blurring parameters are held according to thedistance to the object and the blurring parameter is not limited to alinear function, but may be added a parameter based on the result of anoptical calculation.

Starting the process, the CPU 62a sets pixels on which image processingis to be performed (step S301). Then the CPU 62a obtains the distanceinformation on which coordinate conversion to the distance from theobject plane has been performed (step S302). Then the CPU 62a obtainsthe blurring parameter corresponding to the distance information withreference to the table based on the relation shown in FIG. 19 (stepS303). Then the CPU 62a adds the blurring parameter to the pixels (stepS304). For example, blurring can be performed by low-pass filtering (LPFprocessing) using adjacent pixels, in which the blur amount can bechanged by changing the number of the adjacent pixels. With the averageof all values of pixels to be used equal to the value of a target value,the blur amount increases as the amount of pixels increases.

Then the CPU 62a determines whether or not the processing has beencompleted with all of the pixels (step S305). If the CPU 62a determinesthat the processing has been completed with not all of the pixels, theprocess returns to step S301 to repeat the above-described steps. On theother hand, if the CPU 62a determines that the processing has beencompleted with all of the pixels, the process ends and returns to theprocess shown in FIG. 15.

The basic process of performing imaging, setting the center of theobject plane, manipulating the object plane, and performing the imageprocessing has been described. In any of these steps, an operation canbe undone by selecting “Cancel” or “Back”.

Specifically, as shown in FIG. 20(a), “Cancel” or “Back” is superimposedas an operation button 502a on a display screen 502. The button 502ashould be placed so as not to hide the center of the object plane. Alsothe button 502a is preferably placed so as not to interfere with thecorrection by dragging the rotation center of the object plane.

Also, as shown in FIG. 20(b), an operation button 503a such as “Cancel”or “Back” can be moved by dragging on a display screen 503. Thisprevents the button 503a from interfering with the correction of therotation center of the object plane. Also, the confirmation of theeffect of tilting the object plane can be achieved completely in everycorner by moving the button.

Furthermore, the confirmation of the effect corresponding to “cameramovement (swing, tilt)” may require looking over every corner as well asaround the center of the object plane at one time. In order to do this,as shown in FIG. 20(c), an operation button 504a such as “Hide” may beplaced on a display screen 504. The “Hide” operation button 504a canalso be moved by dragging as shown in FIG. 20(b).

Tapping the “Hide” operation button 504a causes all elements other thanthe display screen 504 to be hidden and allows looking over the screenat one time to confirm the effect of manipulating the object plane.Tapping anywhere on the display screen 504 again causes the hiddenelements to be shown again. Operating the “Hide” operation button 504afrom a menu allows selecting an element not to be hidden and displayingthe element. For example, it is possible to only show the rotationcenter of the object plane and hide the rest.

(Fourth Embodiment)

Next, a fourth embodiment of the invention is described.

The configuration of an image processing apparatus in accordance withthe fourth embodiment is similar to that of the third embodiment (FIGS.13 and 14). Then in the following description, these figures arereferred as appropriate. Also, when referring to the flowcharts shown inFIGS. 15 and 16, features are described with reference to the steps inthese figures.

In order to obtain the effect corresponding to “camera movement (swing,tilt),” manipulating the depth of field from the object plane as well astilting the object plane is desirably possible. In view of the above, inthe image processing apparatus and the image processing method inaccordance with the fourth embodiment, when fixing the rotation centerof the object plane (corresponding to step S204 of FIG. 16), fixing thecenter of the object plane and fixing the depth of field can be achievedby tapping twice the rotation center of the displayed object plane.

Normally, the depth of field has the relation with the aperture as shownbelow.

-   When performing imaging with a bright aperture, the depth of field    is shallow (the shutter speed is fast).-   When performing imaging with a dark aperture, the depth of field is    deep (the shutter speed is slow).    Then, the depth of field is fixed by the time interval of two taps.    When the interval is short, the depth of field is set to be shallow.    When the interval is long, the depth of field is set to be deep.    This would be intuitive for a user who operates a camera, because    this corresponds to the relation with how fast (or slow) the shutter    speed is.

Desirably, the depth of field can be changed even after the rotationcenter of the object plane is fixed or the object plane is tilted. So,in the image processing apparatus and the image processing method inaccordance with the fourth embodiment, the depth of field can be resetby tapping twice the rotation center of the object plane and imageprocessing based on the reset depth of field (corresponding to step S207of FIG. 16) can be performed as many times as needed until finishing therotation of the object plane is requested (corresponding to step S208 ofFIG. 16).

Specifically, as shown in FIG. 21(a), in the display screen 505, whenthe depth of field is set according to the user operation (in thisexample, when the rotation center of the object plane is tapped twice),rough indication of the depth of field is superimposed on the displayscreen 505 as a graph indicator 505a. This allows the user to perform amore intended operation.

Specifically, in FIG. 21(a), when the rotation center of the objectplane is first tapped by the user operation, the graph indicator 505athat varies depending on the interval between the taps is displayed onthe display screen 505 (at the position that does not overlap therotation center of the object plane). In this example, the graphindicator 505a is band graph-like and changes the indication from“Shallow” to “Deep” as time elapses from the first tapping to help theuser set the depth of field.

The graph indicator 505a is not limited to indicating that the depth offield is “Shallow” or “Deep”. As seen from a display screen 506 shown inFIG. 21(b), the graph indicator 506a may indicate the range within whichfocus is achieved between the “Front” and the “Back” of the objectplane. A specific value may also be indicated as well as the intuitivegraph indicator. As an example of such a value, “the depth of fieldcorresponding to F2.8 with the angle of view corresponding to 85 mm by35 mm size conversion” may be indicated. The angle of view by 35 mm sizeconversion may be automatically calculated from the imaging informationof the image data, and the F-number with that angle of view may bedisplayed.

For setting the depth of field by tapping twice the center of the objectplane, tapping once the center of the object plane may be taken as anrequest for finishing the rotation of the object plane. Then, as shownin FIG. 21(c), a “Finish” button 507a is separately superimposed on adisplay screen 507. This “Finish” button 507a may also move to a desiredposition on the display screen 507 by dragging. This prevents the“Finish” button 507a from interfering the correction of the rotationcenter of the object plane. Also, the confirmation of the effect oftilting the object plane can be achieved completely in every corner bymoving the button.

(Fifth Embodiment)

Next, a fifth embodiment of the invention is described.

The configuration of an image processing apparatus in accordance withthe fifth embodiment is similar to that of the third embodiment (FIGS.13 and 14). Then in the following description, these figures arereferred as appropriate. Also, when referring to the flowcharts shown inFIGS. 15 and 16, features are described with reference to the steps inthese figures.

The image processing apparatus and the image processing method inaccordance with the fifth embodiment are characterized in that, insteadof superimposing the “Finish” button on the displayed image, tappingtwice and requesting for finishing the rotation of the object plane areperformed using the same user interface (UI).

Specifically, as shown in FIG. 22(a), on a display screen 508, a firsttapping on the touch panel 76 causes a graph indicator 508a to providean indication for helping the user set the depth of field between“Shallow” and “Deep,” and to display “Finish” on the top of the graphindicator 508a for the user to request to finish the rotation of theobject plane.

Then, as shown in FIG. 22(b), on a display screen 509, the indication ofa graph indicator 509a gradually changes from “Shallow” to “Deep” astime elapses from the first tapping on the touch panel 76, and holds“Deep” for a predetermined period. If a second tapping has beenperformed on the touch panel 76 up to this time, the CPU 62a detectsthat the depth of field has been set, and performs the image processing(corresponding to step S207 of FIG. 16).

If a predetermined time required for the user to set the depth of fieldto “Deep” (for example, one second) has elapsed with no second tappingperformed on the touch panel 76, an indication of a graph indicator 510aon a display screen 510 is gradually changed towards “Finish,” as shownin FIG. 22(c). If the second tapping has been performed on the touchpanel 76 during this period, the depth of field is set to “Deep”.

If the indication has reached “Finish” with no second tapping performedon the touch panel 76, the CPU 62a takes this as a request for finishingthe rotation of the object plane (corresponding to step S208 of FIG.16).

Thus, in the image processing apparatus and the like in accordance withthe fifth embodiment, after a predetermined time elapses, the indicationchanges to that for requesting to finish the rotation of the objectplane (corresponding to step S208 of FIG. 16), which eliminates the needfor displaying “Finish” button, avoiding causing complicated screendisplay. Of course, the depth of field may be set by specifying focallength and aperture by 35 mm size conversion through a menu operation bythe user.

A method for setting the depth of field is not limited to settingaccording to the time length for tapping twice. For example, a firsttapping may cause a display screen 511 shown in FIG. 22(d) to bedisplayed, and a second tapping on the touch panel 76 may be performedon a graph indicator 511a in which the depth of field can be set between“Shallow” and “Deep”, and “Finish” is displayed. With this method, bothsetting the depth of field and requesting finishing the rotation of theobject plane can be performed in one UI without taking time for changingthe indication for requesting finishing the rotation of the objectplane.

Furthermore, when tapping the rotation center of the object plane (forfixing the object plane in step S204 of FIG. 16, setting the depth offield, and requesting to finish the rotation of the object plane), a taprange that is determined to be “the rotation center of the object plane”may be displayed.

Specifically, FIG. 23(a) is an example of displaying the tap range thatis determined to be “the center of the object plane” as an circle 512ain a display screen 512. FIG. 23(b) is an example of displaying the taprange that is determined to be “the rotation center of the object plane”as a rectangle 513a in a display screen 513.

When tapping is performed within this range on the touch panel 76, theCPU62a determines that “the center of the object plane” is tapped andperforms an appropriate processing. When tapping is performed outsidethe range, other processing (such as rotating or tilting the objectplane) is performed.

Again, of course, various screen displays such as “Cancel,” “Back,”“Hide,” and “Finish” (buttons), and rough indication of the depth offield are placed desirably outside the range that is determined to be“the center of the object plane”.

(Sixth Embodiment)

Next, a sixth embodiment of the invention is described.

The configuration of an image processing apparatus in accordance withthe sixth embodiment is similar to that of the third embodiment (FIGS.13 and 14). Then in the following description, these figures arereferred as appropriate. Also, when referring to the flowcharts shown inFIGS. 15 and 16, features are described with reference to the steps inthese figures.

In the image processing apparatus and the image processing method inaccordance with the sixth embodiment, when the object plane is rotated(tilted) according to step S206 of FIG. 16, the angle of the rotation(amount of tilting the object plane) can be specified by tapping(touching) intensity on the touch panel 76, and the tilt amount can bespecified by tap location as well, as shown in FIG. 24.

Specifically, the sixth embodiment helps the user operation by graduallydisplaying the angle of tilting the object plane on a display screen514, as shown in FIG. 24.

The proximity of the center of the object plane is a tap range that isdetermined to be “the rotation center of the object plane”. The furtherfrom there, the larger the amount of tilt (angle of rotating/tilting theobject plane) is specified. In other words, the amount of tilt is setdepending on the distance from the coordinates of the center of theobject plane to the coordinates of the tap position (X, Y). For example,the amount of tilt may be one degree for proximity to the rotationcenter of the object plane, two degrees for far areas, and three degreesfor further areas.

The angle of the rotation, that is the angle of coordinate conversionbased on solid analytic geometry, may be set so as to change graduallyfrom the rotation center of the object plane, and more preferably, canbe precisely adjusted according to the distance from the rotation centerof the object plane to the tap location. Also, even when graduallyindicated, the angle may actually be set more precisely than what isindicated.

Thus the sixth embodiment provides intuitive operation to the user bychanging the tilt angle in proportion to the distance from the rotationcenter of the object plane to the tap location.

The rotation angle of the object plane can be changed according to theangle of view of the captured image (recorded as metadata in step S101of FIG. 15). For example, in actual lenses, the amount of tilt of theobject plane when a lens is tilted varies among 24 mm, 45 mm, and 90 mmlenses. With the same tilt angle between the lens and imaging device(e.g., 10 degrees), the wider angle the lens has, the more the amount oftilt of the object plane is.

(Seventh Embodiment)

Next, a seventh embodiment of the invention is described.

The configuration of an image processing apparatus in accordance withthe seventh embodiment is similar to that of the third embodiment (FIGS.13 and 14). Then in the following description, these figures arereferred as appropriate. Also, when referring to the flowcharts shown inFIGS. 15 and 16, features are described with reference to the steps inthese figures.

In photography, imaging may be performed with the horizontal andvertical directions precisely predetermined. Also, in manipulation ofthe object plane using an actual “camera movement (swing, tilt)” lens,the horizontal and vertical directions may be more preciselypredetermined. In view of the above, the image processing apparatus andthe image processing method in accordance with the seventh embodiment ischaracterized in that, in step S205 of FIG. 16, as shown in FIG. 25(a),a horizontal line 515a and a vertical line 515b are displayed on adisplay screen 515.

The horizontal line 515a and the vertical line 515b may be displayedbased on the position and angle of the camera recorded as metadata (froman output of a gyroscope, for example) in performing imaging (step S101of FIG. 15). More simply, the horizontal line 515a and the vertical line515b may be displayed as image information that passes through therotation center of the object plane. When the displayed horizontal line515a or vertical line 515b are tapped on the touch panel 76, therotation angle of the object plane is fixed by the CPU 62a.

For example, as shown in FIG. 25(b), when a horizontal line 516a on adisplay screen 516 is tapped on the touch panel 76, the CPU 62a limitsthe direction of the rotation of the object plane so that the rotationwill be performed only around the axis on the horizontal line 516a, anddisplays a mark 516c (in this example, triangle) for indicating that therotation axis is limited to the horizontal line 516a.

With this state, the rotation of the object plane is performed only onthe fixed line axis, and then, the angle of the rotation is determinedby the distance from the line coordinate (Y in this case) to the taplocation Y in this case). When the fixed line is tapped again on thetouch panel 76, the CPU 62a releases the line.

As seen from a display screen 517 shown in FIG. 25(c), various lines maybe displayed based on a menu operation without limiting horizontal lineor vertical line. For example, some of actual “camera movement (swing,tilt)” lenses allow determining the angle for each predetermined angle.Such use would be familiar with the user.

(Eighth Embodiment)

Next, an eighth embodiment of the invention is described.

The configuration of an image processing apparatus in accordance withthe eighth embodiment is similar to that of the third embodiment (FIGS.13 and 14). Then in the following description, these figures arereferred as appropriate. Also, when referring to the flowcharts shown inFIGS. 15 and 16, features are described with reference to the steps inthese figures.

The image processing apparatus and the image processing method inaccordance with the eighth embodiment is characterized in that, inrotating (tilting) the object plane (step S206 of FIG. 16), the stateand angle of the object plane is displayed.

For actual “camera movement (swing, tilt)” lenses, the angle with whichthe object plane is tilted can be known only through a finder orcaptured image. On the other hand, in this embodiment, which providessimilar effect by image processing, the angle of rotating the objectplane (corresponding to step S206 of FIG. 15), that is the angle to beused for the coordinate conversion, is recorded to the RAM 62b. Thisallows obtaining information of the amount of tilting the object planefrom the initial position, which is perpendicular to the optical axis ofthe lens of the camera, when converting the distance information of theimage to the information on the distance from the object plane.

Specific examples of the displays are shown in FIGS. 26(a) to (h). Ondisplay screens 518 to 525, status indications 518a, 519a, 520a, 521a,522a, 523a, 524a, and 525a are displayed, respectively, each of whichindicates the state and angle of the object plane. The states of theobject plane indicated by these status indicators 518a, 519a, 520a,521a, 522a, 523a, 524a, and 525a are presented in three dimensions,which would be intuitive and user-friendly.

(Ninth Embodiment)

Next, a ninth embodiment of the invention is described.

The configuration of an image processing apparatus in accordance withthe ninth embodiment is similar to that of the third embodiment (FIGS.13 and 14). Then in the following description, these figures arereferred as appropriate. Also, when referring to the flowcharts shown inFIGS. 15 and 16, features are described with reference to the steps inthese figures.

In the above-described third to eighth embodiments, the object plane isassumed to be a simple plane. However, for the image processingapparatus and the image processing method in accordance with the ninthembodiment, the object plane is not limited to a simple plane. As atypical optical system (lens) has field curvature, the shape of theobject plane for the imaging device, which is typically planar, will notbe planar.

FIGS. 27(a) and (b) are conceptual drawings of such field curvature.FIG. 27(a) shows a meridional image plane. FIG. 27(b) shows a sagittalimage plane. Meridional image plane is also referred to as tangentialimage plane. Sagittal image plane is also referred to as radial imageplane.

Meridional plane is a plane containing an optical axis and a main lightbeam. The meridional image plane is an image plane that gathers the mostmeridional light beams that run in the meridional plane. Sagittal planeis a plane perpendicular to the meridional plane. The sagittal imageplane is an image plane that gathers the most sagittal light beams thatrun in the sagittal plane.

In the ninth embodiment, the shape of the object plane is assumed to becurved like actual lenses, instead of being planer. Accordingly, whenthe coordinate conversion on the distance information (depth map) isperformed so that the rotation center of the object plane will be thecenter of the three-dimensional coordinate system in step S203 of FIG.16 (that is, when the distance information for each pixel of thecaptured image is converted to the information on the distance from theobject plane), the calculation is performed considering the shape of theobject plane.

The shape of the object plane is maintained as data representing therelation between the image height (distance from the center of theimage) and the shape of the object plane. It is maintained as with thedistance information (depth map) so as to represent the relation betweenthe location (X, Y) in the image and the shape of the object plane (Z).

FIG. 28 shows the shape of the object plane represented like the depthmap. In this figure, black represents the front, and white representsthe back. For the shape of the object plane, only the relation betweenthe image height and the shape of the object plane (as shown in FIG. 29)needs to be represented, and maintaining data as much as the distanceinformation is not required.

Data for regions into which the whole image is horizontally andvertically divided may be maintained, or only the relation between theimage height and the shape of the object plane may be maintained. Theshape of the object plane may be maintained and recorded in the RAM 62b,or may be obtained from the outside through the external I/F 69.

Specifically, the CPU 62a performs the following calculations.

(1) First, the values of the coordinates (X, Y, Z) of the pixel set tothe center of the object plane is subtracted from the coordinateinformation of all pixels so that the coordinates (X, Y, Z) will be theorigin (0, 0, 0) of the three-dimensional coordinate system. Thisconverts the information on the distance from the camera to theinformation on the distance from the plane that passes through thecoordinates of the center of the object plane and perpendicular to theoptical axis of the lens used for the imaging.

(2) Then, the shape of the object plane is added to or subtracted fromthe coordinate information (X, Y, Z) of each pixel location in the image(the order of this subtraction may be reversed depending on how to holdthe data of the shape of the object plane). This converts theinformation on the distance from the plane that passes through thecoordinates of the center of the object plane and perpendicular to theoptical axis of the lens used for the imaging to the information on thedistance from the object plane.

Including the calculation of the shape of the object plane increases aprocessing load. However, in order to allow the user to confirm theeffect of the image processing and in order to help comfortable useroperation, it is preferable to calculate the shape of the object planeat this stage. The calculations (1) and (2) can be performed at the sametime.

Two types of calculation methods may be used for the calculation (2).

Align the rotation center of the object plane designated in step S201 ofFIG. 16 (or corrected in step S204) over the center of the shape of theobject plane, and proceed the process.

Absolutely align the center of the image over the center of the shape ofthe object plane, and proceed the process.

In actual optical systems, the center of the image is aligned over thecenter of the shape of the object plane as with the latter. So theeffect given by the latter is preferable to the user. Then the followingdescription is based on the latter. However, in order to meet the needfor a different effect than actual lenses, aligning the rotation centerof the object plane over the center of the shape of the object plane maybe allowed by specifying through a menu operation.

In actual “camera movement (swing, tilt)” lenses, when camera movementis used, the rotation center of the object plane moves away from thecenter of the image depending on the focal length, the tilt angle of theobject plane, and the imaging distance (that is, the distance to theobject). Preferably, the embodiment that provides the “camera movement(swing, tilt)” effect by the image processing also provides a means forallowing the reproduction of such behavior.

Thus, when the object plane is rotated in step S206 of FIG. 16, therotation center of the object plane is moved away from the center of theimage depending on the angle of view of the image (recorded as metadatain step S101 of FIG. 15) and the rotation angle of the object plane.

When the object plane is rotated in step S206 of FIG. 16, the shape ofthe object plane is subtracted from or added to the coordinateinformation (X, Y, Z) of each pixel location in the image (this may bereversed depending on how to hold the data of the shape of the objectplane). This converts the information on the distance from the objectplane to the information on the distance from the plane that passesthrough the coordinates of the center of the object plane andperpendicular to the optical axis of the lens used for the imaging. Inother words, the object plane is temporarily restored to the originalplanar shape.

Then, by the coordinate conversion based on solid analytic geometry, thedistance information is converted to the information on the distancefrom the plane that passes through the coordinates of the center of theobject plane and is rotated by the angle specified/operated by the userfor each pixel. In other words, the coordinate conversion for rotationis performed in the state of being planar.

The relation between the rotation angle of the object plane and theamount of the movement of the rotation center of the object plane fromthe center of the image may be maintained and recorded in the ROM 62b inadvance, or may be obtained from the outside through the external I/F69. The CPU 62a maintains this relation so that the amount of coordinatemovement (X, Y) from the center of the image can be known from therotation angle of the object plane.

The shape of the object plane that is moved (X, Y) from the center ofthe image depending on the rotation angle of the object plane is addedto or subtracted from the coordinate information (X, Y, Z) of each pixellocation in the image (this maybe reversed depending on how to hold thedata of the shape of the object plane). This converts the information onthe distance from the plane that passes through the coordinates of thecenter of the object plane and is rotated by the anglespecified/operated by the user to the information on the distance fromthe object plane. In other words, finally, the object plane is restoredto its shape again.

However some (more than a few) users would need to manipulate the shapeof the object plane. So, it is also preferable to provide a capabilityof changing the shape of the object plane. Then, in the image processingapparatus and the image processing method in accordance with thisembodiment, when the center of the object plane is kept pressed (withouttapping and dragging) in step S204 or S206 of the FIG. 16, as shown inFIG. 30(a), the CPU 62a controls so as to display a display frame 526afor the current shape of the object plane on a display screen 526 of theLCD 68.

In the display screen 526 shown in FIG. 30(a), the display frame 526afor the shape of the object plane can be moved by, for example, draggingthe display frame. When the shape of the object plane displayed on thedisplay frame 526a is tapped or dragged, the CPU 62a changes the shapeof the object plane. The result of changing the shape of the objectplane can be quickly confirmed by the user by performing the imageprocessing in step S205 or S207 of FIG. 16.

The operation of displaying the shape of the object plane is not limitedto keeping the center of the object plane pressed. “Shape of the objectplane” button may also be displayed on the display screen 526. The“Shape of the object plane” button is placed desirably outside the rangethat is determined to be “the center of the object plane”.

Also, a first tapping on the center of the object plane may cause ascreen 527 to be displayed as shown in FIG. 30(b) and a second tappingmay make a graph indicator 527a, displaying the setting of the depth offield from “Shallow” to “Deep,” “Finish,” and “Shape of the objectplane,” available for input relating the depth of field. This allowssetting the depth of field, requesting finishing the rotation of theobject plane, and requesting displaying the shape of the object plane inone UI.

Furthermore, including buttons such as “Cancel” and “Hide” in the screendisplay caused by the first tapping allows manipulating the object planein the same UI.

(Tenth Embodiment)

Next, a tenth embodiment of the invention is described.

The configuration of an image processing apparatus in accordance withthe tenth embodiment is similar to that of the third embodiment (FIGS.13 and 14). Then in the following description, these figures arereferred as appropriate. Also, when referring to the flowcharts shown inFIGS. 15 and 16, features are described with reference to the steps inthese figures.

The rotation center of the object plane and the like may be designatedby a means other than designation by the user. For example, the centerof the object plane and the rotation angle of the object plane may befixed based on a specific point (characteristic point) detected using adetection technique such as “facial recognition”.

Then in the image processing apparatus and the image processing methodin accordance with the tenth embodiment, the CPU 62a recognizes theobject by extracting specific points based on the brightness andcontrast in the image, and matching the positions (positional relation)of the specific points with the pattern stored in the RAM 62b.

In an example shown in FIG. 31(a), a face is recognized by thepositional relation between his/her both eyes, nose, and mouth, and thecenter of object plane is set to the center of the face. In the screen528, the recognition area of the face is indicated by a rectangularframe 528a.

Also, as shown in FIG. 31(b), by recognizing two or more objects in ascreen 529, the object plane and the depth of field can be set so as tocontain the detected specific points (e.g., eyes, noses, and mouths) ofthe objects or object areas (e.g., faces). In the screen 529, therecognition areas of two or more faces are indicated by rectangularframes 529a to 529d.

Furthermore, as shown in FIG. 32(a), for example, when two object areas(e.g., face) are detected, the CPU 62a places the rotation axis of theobject plane on the line connecting the two point on a display screen530 using the coordinate conversion based on solid analytic geometry,and displays the rotation axis 530a to allow the user to performrotation. Then the CPU 62a displays an image blurred according to therotation of the object plane, as shown in FIG. 32(b), on a displayscreen 531.

Furthermore, as shown in FIG. 33(a), for example, when three objectareas (e.g., face) are detected, the CPU 62a displays two or more facerecognition areas as rectangular frames 532a to 532c on a display screen532, and places the object plane on a plane defined by the three pointsusing the coordinate conversion based on solid analytic geometry.

At this point, the CPU 62a may determine an object having the highestproportion of a screen 532 to be the main object, and set the rotationcenter of the object plane to the center of the detected area of themain object. Then, from this point, the user can manipulate the objectplane and set the depth of field. Also, when the detected area isdesignated, the CPU 62a can select the main object (e.g., thecoordinates of the center of the detected area of the object) withinthat area.

For example, as shown in FIG. 33(b), when, in a display screen 533, anobject O₂ located in the back is designated by user operation ratherthan an object O₁ having a higher proportion of the object, the CPU 62aselects the object O₂ (e.g., the coordinates of the center of thedetected area of the object O₂) as the main object and sets the rotationcenter of the object plane to the object O₂.

Furthermore, as shown in FIG. 33(c), when, in a display screen 534, fourobject areas (e.g., face) are detected, the CPU 62a sets the objectplane/depth of field that contains the coordinates of the centers of allof the detected areas using the coordinate conversion based on solidanalytic geometry. The CPU 62a set the object plane so that thecoordinates of the centers of the detected areas will be equidistantfrom the object plane (however, some of the coordinates of the centersof the detected areas may not be on the object plane). The depth offield is set so as to be equal to the distance from the object plane tothe coordinates of the center of each detected area.

In this case, the rotation center of the object plane (534a) is set atthe position being at the minimum distance from the set object plane, asshown in FIG. 33(c), instead of one of the coordinates of the centers ofthe detected areas. Even in this case, according to further designationamong the detected areas, the CPU 62a can also select the main objectand reset the rotation center of the object plane.

Furthermore, as shown in FIG. 34(a), in a display screen 535, detectedobject areas (e.g., face) are indicated by rectangular frames 535a to535d. For these object areas, when selecting a new object or deselectingan already recognized object is performed through a user operation, theCPU 62a resets an object to be used for setting the object plane. In theexample of FIG. 34(b), among the objects displayed on a display screen536, the detected areas of the objects O₃ and O₄ (rectangular frames536c and 536d) are deselected.

Then the CPU 62a specifies the main object (coordinates of the center ofthe detected area of the object) among the reset objects, and displays ablurred image as shown in FIG. 34(c) on a display screen 537.

Of course, the image processing apparatus and the image processingmethod in accordance with the above-described fourth to tenthembodiments can be implemented with any combination of theseembodiments. Also, though the description has been made on the premisethat the touch panel 76 is used, the pointing device used foroperational input is not limited to this, and similar convenience can beprovided to the user, for example, by moving a cursor with thecross-shaped key 77 and specifying with the enter key 74.

The first to tenth embodiments of the invention has been described.These embodiments provide the following advantages.

Specifically, providing an image given the effect corresponding to“camera movement (swing, tilt)” by performing image processingeliminates the need for designing a special optical system andlens-barrel.

Eliminating the need for providing a complicated mechanism (inaccordance with what is called Scheimpflug principle) eliminates theneed for designing a special optical system intending “camera movement(swing, tilt)”. Further, an image processing apparatus that can providean image given the effect corresponding to “camera movement (swing,tilt)” using an image imaged by a compact and low-cost imaging apparatuscan be provided.

Providing an image given the effect corresponding to “camera movement(swing, tilt)” by image processing eliminates the need for complicatedoperation before imaging. This reduces the risk of missing a photoopportunity.

The image processing apparatus can provide a method for allowingflexible manipulation of the object plane corresponding to “cameramovement (swing, tilt)” manipulation using a specially designed opticalsystem and lens-barrel based on a prior art.

For manipulating the object plane, the image processing apparatus canprovide an operational environment that is intuitive and friendly to theuser. For example, the touch panel allows a more intuitive operation.

Displaying the rotation center of the virtual object plane set by theuser on the display screen allows the user to manipulate the objectplane and recognize the state of the object plane more easily.

Displaying the state of operating the virtual object plane by the useron the display screen allows the user to manipulate the object plane andrecognize the state of the object plane more easily.

Even for an image with a deep depth of field imaged by a compact imagingapparatus employing the combination of a small imaging device and ashort focal length lens, the image processing that provides a virtuallyshallow depth of field can be performed by allowing setting a virtualdepth of field.

Allowing setting the shape of the virtual object plane can provide theuser with a wider variety of image creation capability.

It should be noted that the invention is not limited to theabove-described first to tenth embodiments and that various improvementsand modifications can be made without departing the spirit of theinvention. For example, the invention can be implemented as a program ofthe processing used in the above-described image processing apparatus, arecording media on which the above program is recorded, a program forimplementing the above-described image processing method, and arecording media on which the above program is recorded.

The invention claimed is:
 1. An image processing apparatus comprising:an image sensor that captures an image; an object plane setting meansfor setting an object plane in said captured image based on distanceinformation corresponding to a point position designated by a user in adisplayed image based on image data of the captured image; an objectplane changing means for changing the object plane by rotating theobject plane at a rotational angle specified by the user with the pointposition maintained in the object plane; a distance informationconversion means for converting the distance information for at leastone position of the image data external to the changed object planeaccording to the changed object plane using the rotational angle; and animage processing means for performing image processing on the image databased on the converted distance information, wherein the imageprocessing comprises: blurring the image data not on the changed objectplane with a blur amount according to the converted distanceinformation, and not blurring the image data on the changed objectplane,; and whereina display control means for displaying a graphicalindication of the changing of the object plane is displayed on a displayscreen with the image data.
 2. The image processing apparatus accordingto claim 1, wherein the distance information conversion means convertsis further configured to convert the distance information for each pointposition of the image data based on the coordinates of the designatedpoint position.
 3. The image processing apparatus according to claim 1,wherein the object plane changing means changes is further configured tochange the object plane by rotating the object plane on the designatedpoint position as rotation center.
 4. The image processing apparatusaccording to claim 3, wherein, when the object plane is changed by theobject plane changing means, the distance information conversion meansconverts is further configured to convert the distance information foreach point position of the image data based on the coordinates of thedesignated point position and the angle and a direction of the rotation.5. The image processing apparatus according to claim 1, wherein theimage processing means blurs is further configured to blur the imagedata such that the blur amount increases as the distance from the objectplane increases.
 6. The image processing apparatus according to claim 1,wherein the image processing means blurs is further configured to blurthe image data so that points positions within a predetermined distancefrom the object plane will be are blurred with the a same blur amount.7. The image processing apparatus according to claim 1, wherein theimage processing means adjusts is further configured to adjust the bluramount based on depth of field information indicating a depth of field.8. The image processing apparatus according to claim 1, furthercomprising: a storage means for storing the image data and the distanceinformation in association with each other; a designation means fordesignating the point position; and a setting means for setting a tiltangle and a tilt direction of the object plane containing the pointposition with respect to an optical axis of a lens.
 9. The imageprocessing apparatus according to claim 8, wherein, when the pointposition is designated by the designation means and the tilt directionand the tilt angle of the object plane with respect to the optical axisof the lens is set by the setting means, the object plane setting meanssets the object plane based on the tilt direction and the tilt angle.10. The image processing apparatus according to claim 8, furthercomprising a ranging means for ranging distance, wherein distanceinformation obtained by the ranging means is stored by the storage meansin association with the image data.
 11. The image processing apparatusaccording to claim 8, wherein the designation means is a touch panel,and, when a point position is designated in the captured image, thetouch panel accepts the a location designation by detecting thecoordinates.
 12. The image processing apparatus according to claim 11,wherein the location designation by the designation means is performedby touching the touch panel a predetermined number of times.
 13. Theimage processing apparatus according to claim 8, further comprising awherein the display control means is for displaying the a locationdesignated by the designation means on the a display screen along withthe captured image, wherein the designation means accepts the change ofthe location based on the dragging of the designated point position. 14.The image processing apparatus according to claim 8, wherein thedesignation means and setting means are a touch panel, and whereindesignating at least one point position in the captured image, changingthe a designated location, and setting the tilt direction and the tiltangle of the object plane containing the designated point position withrespect to the optical axis of the lens are performed by touching thetouch panel a number of predetermined times.
 15. The image processingapparatus according to claim 1, wherein the distance informationconversion means converts the distance information for each given regionincluding two or more pixels in the image data according to the objectplane.
 16. An image processing method comprising the steps of: capturingan image with an image sensor; an image processor setting an objectplane in said captured image based on distance information correspondingto a point position designated by a user in a displayed image based onimage data of the captured image; changing the object plane by rotatingthe object plane at a rotational angle specified by the user with thepoint position maintained in the object plane, in which the rotationalangle is with respect to the object plane; converting the distanceinformation of the image data according to the changed object plane,wherein the converting the distance information of the image dataaccording to the changed object plane includes, for a given point Pn onthe object plane having coordinates Xn, Yn and Zn, subtractingcoordinates of the position designed by the user from the coordinatesXn, Yn and Zn of the given point Pn to obtain coordinates Xn′, Yn′ andZn′, and subtracting, from a first coordinate of the coordinates Xn′,Yn′ and Zn′, a product of a second coordinate of the coordinates Xn′,Yn′ and Zn′ and a tangent of the rotational angle, the first coordinatebeing different from the second coordinate; and performing imageprocessing on the image data based on the converted distanceinformation, wherein the image processing comprises: blurring the imagedata of an object not on the changed object plane with a blur amountaccording to the converted distance information, and not blurring theimage data of an object on the changed object plane,; andwhereindisplaying a graphical indication of the changing of the objectplane is displayed on a display screen with the image data.
 17. An imageprocessing program stored on a A non-transitory computer-readable mediumon which is stored an image processing program, the program causing acomputer to execute an image processing method comprising the steps of:capturing an image with an image sensor; setting an object plane in saidcaptured image based on distance information corresponding to a pointposition designated by a user in a displayed image based on image dataof the captured image; changing the object plane by rotating the objectplane at a rotational angle specified by the user with the pointposition maintained in the object plane, in which the rotational angleis with respect to the object plane; converting the distance informationof the image data according to the changed object plane, wherein theconverting the distance information of the image data according to thechanged object plane includes, for a given point Pn on the object planehaving coordinates Xn, Yn and Zn, subtracting coordinates of theposition designed by the user from the coordinates Xn, Yn and Zn of thegiven point Pn to obtain coordinates Xn′, Yn′ and Zn′, and subtracting,from a first coordinate of the coordinates Xn′, Yn′ and Zn′, a productof a second coordinate of the coordinates Xn′, Yn′ and Zn′ and a tangentof the rotational angle, the first coordinate being different from thesecond coordinate; and performing image processing on the image databased on the converted distance information, wherein the imageprocessing comprises: blurring the image data of an object not on thechanged object plane with a blur amount according to the converteddistance information, and not blurring the image data of an object onthe changed object plane,; and whereindisplaying a graphical indicationof the changing of the object plane is displayed on a display screenwith the image data.
 18. An image processing apparatus comprising: animage sensor that captures an image; a processing device that receivesthe image captured by the image sensor; and a memory includinginstructions which, when executed by the processing device, control:setting an object plane in said captured image based on distanceinformation corresponding to a position designated by a user in adisplayed image based on image data of the captured image; changing theobject plane by rotating at a rotational angle specified by the userwith the position maintained in the object plane in which the rotationalangle is with respect to the object plane; converting the distanceinformation of the image data according to the changed object plane,wherein the converting the distance information of the image dataaccording to the changed object plane includes, for a given point Pn onthe object plane having coordinates Xn, Yn and Zn, subtractingcoordinates of the position designed by the user from the coordinatesXn, Yn and Zn of the given point Pn to obtain coordinates Xn′, Yn′ andZn′, and subtracting, from a first coordinate of the coordinates Xn′,Yn′ and Zn′, a product of a second coordinate of the coordinates Xn′,Yn′ and Zn′ and a tangent of the rotational angle, the first coordinatebeing different from the second coordinate; image processing on theimage data based on the converted distance information, wherein theimage processing comprises: blurring the image data of an object not onthe changed object plane with a blur amount according to the converteddistance information, and not blurring the image data of an object onthe changed object plane; and displaying a graphical indication of thechanging of the object plane with the image data.